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SYSTEM AND METHOD FOR ENHANCING LIVE SPEECH WITH 
INFORMATION ACCESSED FROM THE WORLD WflDE WEB 



Field of Uie invention 

The present invention relates generally to techniques for integrating in a live 
5 event, means for accessing complementary infomiatlon. and more particularly to 
systems and methods for enhancing a live event, such as a live speech or a live 
broadcast program, with information, in particular with information that can be 
accessed from the World Wide Web. 

Bacicground of ttie invention 

10 Need for Information 

In our society, people have a growing need for information and services Most 
spectators attending to a live event (e.g.. as auditors of a conference given by a 
speaker), or auditors attending to a live radio or television broadcast program, want 
to have access to complementary infomiatlon. This complementary infomiation may 
15 consist in the biography of a player of a football match, the historical background on 
events crted on a news program, or athletic records during the transmission of an 
Olympic competition. 

In fact, today people are looking for more information about what they are 
hearing or listening locally, as spectators of live events, or remotely, as auditors of 
20 live broadcast programs : 

• Consumers want to have access to special seivices associated with 
advertised products. 

• Media providers expect new sources of profit by extending the quantity and 
quality of services and information provided to consumers and more 
particularly to auditors of live television or radio programs. 

• Advertisers are looking for new and more effective forms of advertisement. 



Independently of the massive development of radio and television, on-line 
services such as those provided on the World Wide Web (i.e., the Web), have 
rapidly emerged in our society and are now widely available. Such on-line services 
based on the Internet technology, provide access to a huge amount of information 
on an interactive basis. The Internet is a global network of computers. The Internet 
connects computers based on a variety of different operating systems or languages 
using a language referred to as TCP/IP (Transmission Control Protocol/Internet 
Protocol"). With the increasing size and complexity of the Internet, tools have been 
developed to help users to find information they need on the network. These tools 
are often called "navigators" or "navigation systems". The World Wide Web ("WWW" 
or "the Web") Is a recent superior navigation system. The Web is : 

• an Internet-based navigation system, 

• an information distribution and management system for the Internet, and 
15 •a dynamic format for communicating on the Web. 

Internet and the Web fs transforming our society by offering to millions of 
users, the opportunity to access and exchange Information and to communicate 
between each other. By integrating images, text, audio and video, a user on the 

20 Web using a graphical user interface can today transparently communicate with 
different computers on the system, different system applications, and different 
Information formats for files and documents Including, for example, text, sound and 
graphics. Currently, on-line systems on the Web offer a variety of different 
services to users, for instance, private message services, electronic commerce, 

25 news, real-time games, access to electronic databases, electronic newsletters, 
business-to-business transactions, or job placement services. 

But, even if such on-line services are now available, the searching and finding 
of the relevant information on the Web remains an arduous task, sometimes taking 
hours, even for experienced users. Obviously, since the Web is essentially an open, 
30 multi-point to multi-point network, each user can select and retrieve different 
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information from many different senders. In fact, today, most on-line interactions wrth 
the Web occur merely through textual inputs for Instance by entering URLs (Uniform. 
Resource Locator) names, by entering l<ey words on search tools, or by activating 
textual hyperlinks on HTML (Hypertext IVIarkup Language) documents. Even if In a 
5 near future, the development of audiovisual interfaces (e.g.. human speech 
interfaces. Web-phone integration) will render textual inputs less and less dominant 
in on-line environments, there is a good chance that the Web remains user 
unfriendly due to its massiveness, its lack of organization, and its randomness. 
Simply stated. In the Web, there Is no order or direction. Information remains most of 
10 the time hard to find and even worse, In a foreseeable future, it will remain a difficult 
task to find the required Information Into the desired context. 

On-line Services from Live Speech 

Unlike the multi-point to multi-point Web network, a ; live speech to an 
audience (the audience being In the same location as the speaker or the audience 

15 being remotely located, i.e.. accessed through a radio or television broadcast 
station) is primarily a communication from a single emitter to multiple receivers. 
Every auditor receives the same content, locally from the speaker, or remotely 
through the broadcasting station. 

Thus, to provide on-line services similar to those that can be accessed on the 

20 Web, a first problem in a live speech is that the information flows continuously in the 
same direction, from a single source to multiple receivers, from a provider to multiple 
auditors. The communication flow Is limited to one direction without any exchange of 
Information with the auditors. People cannot directly interact with. the oral Information 
that is received, to access additional infomnation or services. 

25 Moreover, when people hear a live speech, a problem for the auditors is to 

select topics of interest and then to Identify the network addresses (I.e., URLs) to 
access (e.g., from the Web) the multimedia information or services related with the 
selected topics. Until today, this problem has been partially solved. 



FR9-2002-0065 

4 




To provide web-IIke capabilities to the oral or radio Information, a solution is to 
embed information (e.g., URLs) into tlie transmitted broadcast audio signals or on 
separate channels (simultcast). Examples of such systems are described In the 
following patents : US patent 6,125,172 entitled "Apparatus and method for Initiating 

5 a transaction having acoustic data receiver that filters human voice", US patent 
6.098,106 entitled "Method for controlling a computer with an audio signal", US 
patent 5,841,978 entitled "Networt^ linking method using steganographically 
embedded data objects", US patent 5,832,223 entitled "System, method and device 
for automatic capture of Internet access information in a broadcast signal for use by 

10 an Intemet access device". US patent 5,761,606, entitled "Media online sen/ices 
access via address embedded in video or audio program". US patent 5,189,630 
entitled "Method for encoding and broadcasting infomnation about live events using 
computer pattern matching techniques", US patent 5,119.507 entitled "Receiver 
apparatus and methods for identifying broadcast audio program selections in a radio 

15 broadcast system" or US patent 6,061,719 entitled "Synchronized presentation of 
television programming and web content". 

The systems and methods described in these patents require the 
transmission of a complementary Information (e.g., URLs) encoded, embedded or 
modulated on the same audio or video signal or transmitted on a separate channel, 
20 concurrently with the transmission of the main program. Radio or television stations 
must comprise means for encoding, modulating and transmitting along with the 
audio signal this complementary information. The radio auditors or television viewers 
must be equipped with special receivers and decoder circuits for recovering this 
information. 

25 Independently of the herein above discussed an-angements, systems have 

been developed to enable auditors to "pre-se/ecr topics of interest (i.e., keywords or 
sentences) and to associate these topics with pre-speclfied networi< addresses (i.e., 
URLs). These pre-specified network addresses are used to access multimedia 
Information or services related with the pre-selected topics. In general temns, all 

30 these system are based on speech recognition techniques. These techniques are 
used to identify keywords (i.e., selected words or sentences) for performing specific 
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actions in response to tlie recognition of specific sounds. Examples of these 
systems can be found in tlie following patents : 

US patent 5,946.050 entitled "Keyword listening device" discloses a method 
and a system for monitoring the audio portion of a broadcast signal by means of a 
5 keyword listening device, where a relatively limited set of keywords are stored. The 
keyword listening device monitors the broadcast signal for any of these keywords. 
Upon recognition of any one or more of the keywords, the broadcast audio signal is 
recorded for a period of time and then fully analyzed. After analysis, and In 
dependence upon the recorded and analyzed broadcast audio signal, a number of 
10 different functions, such as connection to an external network at a specified address, 
or control of a video cassette recorder, may be performed. 

US patent 6,011.854 entitled "Automatic recognition of audio information In a 
broadcast program" discloses an audio processing system to search for information 
reports or updates (such as traffic, weather, time, sports, news, and the like) 

15 broadcast over one or several radio stations. The search is based on at least one 
keyword (such as "traffic", "weather", "time", "sports", "news" depending on the 
desired report) pre selected by the user, and entered Into the audio processing 
system. While speech recognition software used by the audio processing system 
scans the radio station for the requested information report, the user may listen to 

20 other audio sources (a CD, a tape, another radio station broadcast, etc.) without 
being required to monitor (that is. listen to) the information content from those audio 
sources. Once the requested information report Is detected based on the entered 
keyword used in the radio broadcast, the audio processing system switches its audio 
output to the radio station transmitting the desired broadcast, so that the user can 

25 timely and conveniently listen to the traffic,, weather, time, sports, and/or news 
reports or updates. 



US patent 6,332,120 entitled "Broadcast speech recognition system for 
keyword monitoring" discloses a system where broadcast audio is automatically 
monitored for infomnation of interest. This system comprises a computer processor 
with a memory for storing a vocabulary of keywords of interest., an audio receiver for 
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receiving an audio broadcast and a speech recognition system, associated witli the 
audio receiver and the computer processor, for detecting when one of the keywords 
appears in a received audio segment. A report generator associated with the 
computer processor and responsive to the detection of a keyword, generates a 
5 report with details related to the detected keyword and its context. 

Even if the systems previously mentioned don't require the transmission of 
complementary infonnation embedded with the audio signal (or on a secondary 
signal concurrently with the retransmission of the main program), the auditors must 
be equipped with receivers with speech recognition capabilities to detect the 
10 occurrence of hyperlinked temns in the data stream. 

In the field of speech processing, the ability to identify occun-ences of words 
or sentences in a stream of voice data is commonly called "word spotting". The goal 
of audio wordspotting is to identify the boundaries of a search term within a digitized 
continuous speech stream without prior manual transcription. Searching and 

15 indexing a live speech that may be pronounced by any speaker, is particularly 
problematic. This is due in large part, to the limited capabilities of the existing 
automatic speech recognition technology. It is important to note that in the above 
discussed systems, the word spotting task is done on the auditor side, in a speaker 
independent manner, with unrestricted vocabulary, and employ speech models 

20 trained using voice data other than the data to recognize. 

In fact, a fundamental problem with all systems, is the unreliable behavior of 
the state-of-the-art speech recognition technology for performing 'Sword spotting" 
(i.e.. identification of pre-specified keywords or tenms) on a continuous manner, 
independently to the speaker, based on unknown or generic speaking styles. 
Zz vocabularies, noise levels and language models. 

As shown in the foregoing discussion, even if during these last years, 
Ir.ta.'-active systems have been developed for increasing and improving the level of 
interaction with users and for providing more information and more learning or 
eatertainment opportunities (e.g.. interactive television. WebTV), important sources 
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of information, sucli as those that can be found in the Web, still remain inaccessible 
for auditors of a live speech (e.g., a live conference or a live Interview received from 
a radio or television broadcast). 

Therefore, today there Is a need to provide a convenient, universal, and easy 
5 mechanism for enabling people attending a live speech (e.g., or receiving a live 
broadcast program) to select and access complementary information. 

There is also a need for speakers and producers of live broadcast programs 
to create hyperlinks from selected terms (generally selected spoken utterances, 
words or sentences) intended to be pronounced during, a speech (e.g.. on the 
10 course of a conference or during a live radio or television program), to relevant data . 
on the Web without embedding these hyperlinks in conventional one-way broadcast 
signals, and more generally without physically transmitting . these hyperlinks and 
without modifying conventional transmitters or receivers. 

Objects of the Invention 

15 It is a broad object of the invention to enhance audio infomiatlon such as a 

live speech or a live radio or television broadcast program, with complementary 
information or services related to this audio information. 

It is another object of the invention to create hyperlinks between selected 
terms or words intended to be pronounced by a speaker in the course of a speech 
20 and complementary infomiatlon related to these selected terms. 

It is another object of the Invention to identify hyperlinked terms as they are 
pronounced by a speaker in the course of a speech, and to activate the hyperiinks 
that are associated with the identified temis. 



It is another object of the invention to enable an auditor to select during the 
of a live speech, one or several terms related to topics drawing his attention 
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to 



the topics previously selected. 



It is a further object of the Invention to enable an auditor of a live speech to 
access information related to topics that have drawn his attention during the speech 
5 with the minimum effort and involvement from his part while minimizing the 
complexity of the required equipment. 



Summary of the invention 



The present Invention Is directed to a system, method and computer program 
as defined In independent claims for enabling an auditor of a live speech to access 
10 immediately after or at a later time, complementary information related with temns 
pronounced during the speech. 

The system associates hyperilnks (i.e., URLs) with selected terms or words 
likely to be pronounce by the speaker in the course of the speech. A speech 
recognition system operating on a speaker device (i.e., a computing system a with a 
15 microphone connected to it) recognizes during the speech (i.e., word spotting) the 
pronunciation by the speaker of anyone of said hyperlinked terms, and records the 
time at which each recognized hyperlinked temn has been pronounced. 



The system is also based on the synchronization of the speaker device with 
several auditors devices (e.g., workstations, portable computers, personal digital 

20 assistants - PDAs, smart phones, or any other type of hand held computing devices) 
according to a same universal time, so that the flow of infomnation transmitted by the 
speaker and received by the auditors is always synchronized, independently of the 
relative positions of the speaker and auditors. Each time the auditor perceives an 
interesting topic during the speech, he immediately selects the topic simply by 

25 pressing a reserved key on the auditor's device. Universal times at which topics are 
selected by the auditor are stored in the auditor device. 
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In a preferred embodiment of the Invention, the synchronization between 
speaker and auditor's devices is done referring to an universal time such as the 
Global Positioning System Time (GPS-time). the Global Orbiting Navigational 
Satellite System (GLONASS) time or another suitable universal . time based on a 
5 satellite system. A GPS or GLONASS receiver Is Integrated or connected to the 
speaker device. A GPS or GLONASS receiver Is integrated or connected to each 
auditors device. Each auditor device is independent and separate from the radio or 
television sets used by the auditor to receive the broadcast speech. .; 

More particularly, the present invention discloses a system, method and 
10 computer program for generating from a speaker device, a Speech Hyperlink-Time 
table during a speech, said Speech Hyperlink-Tlme table being accessible by one or 
a plurality of auditors. The method comprises the steps of: 

• locating and accessing a Speech Hyperlink table comprising : . .. 

• means for identifying one or a plurality of predefined hyperiinked terms. 
15 '"tended to be pronounced by a speaker; 

• means for locating and accessing information related to each of said one or 
plurality of predefined hyperiinked terms; 

• retrieving means for locating and accessing a Speech Hyperiink-Time table; . 

20 during the speech : 

• recognizing, when they are pronounced by the speaker, the hyperiinked terms 
predefined in the Speech Hyperlink table by means of a speech recognition 
system connected to the speaker device; 

for each recognized hyperiinked term : 



25 



detemiining an universal-time corresponding to the pronunciation by the speaker 

of the recognized hyperiinked term; 

creating a new record on the Speech Hyperiink-Time table; 

copying into said new record : 
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• the universal-time corresponding pronunciation by the speal^er of the 
recognized hyperlinked term; 

• means, retrieved from the Speech Hyperlinl< table, for identifying the 
recognized hyperllnl<ed term; 

5 • means, retrieved from the Speech Hyperlinl< table, for locating and accessing 
information related to the recognized hyperlinked term. 

The present invention also discloses a system, method and computer 
program for selecting from an auditor device, one or a plurality of hyperlinked temns 
pronounced by a speaker during the course of a speech, and for accessing, at a 
10 later time, information related to each of said one or plurality of selected hyperlinked 
terms. The method comprises the steps of : 

• each time a selection command Is received for selecting a term currently 
pronounced by a speaker during the course of a speech: 

• determining the current universal-time; 

15 • recording the current universal-time in a Selections Hyperiink-Time table; 

• accessing a Speech Hyperiink-Time table, said Speech Hyperiink-Time table, 
accessible by one or a plurality of auditor devices, comprising for each one of a 
plurality of predefined hyperiinked terms pronounced by the speaker : 

• an universal-time interval con-esponding to the pronunciation of the 
20 predefined hyperiinked term; 

• means for identifying the predefined hyperiinked terni; 

• means for locating and accessing information related to the predefined 
hyperiinked term; 

. for each. universal-tlm.e recorded. in the Selections. Hyperlink-Time table: 

25 • identifying in the Speech Hyperiink-Time table, the selected hyperiinked term 
pronounced at the recorded universal-time; 

• retrieving from the Speech Hyperiink-Time table, said means for identifying and 
for locating and accessing the selected hyperiinked term; 
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• Storing the retrieveJSeans for Identifying the selected hyperllnked term and for 
locating and accessing Infomiation related to the selected hyperllnked term into 
the Selections Hyperiink-Time table. 

Further embodiments of the Invention are provided in the appended 
5 dependent claims. 

The foregoing, together with other objects, features, and advantages of this 
invention can be better appreciated with reference to the following specification 
claims and drawings. 
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Br/ef description oftiie drawings 



The novel and inventive features believed characteristics of the invention are 
set forth in the appended claims. The Invention itself, however, as well as a 
preferred mode of use. further objects and advantages thereof, will best be 
understood by reference to the following detailed description of an illustrative. 
15 detailed embodiment when read in conjunction with the accompanying drawings 
wherein: ' 



• Figure 1 shows how an auditor of a live speech such as a conference perceives 
topics of Interest. 

• Figure 2 shows how an auditor perceives topics of interest when listening to a 
20 live radio or television program. 

• Figure 3 Illustrates how.- according to the present invention, the speaker 
workstation and auditors devices are synchronized according to a same universal 
time, and how a speech recognition system operating on the speaker device, 
recognizes hyperlinked tenns pronounced by the speaker. 



25 



Figure 4 shows how. according to the present invention, the speaker prepares 
in advance the text of the speech. 
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• Figure 5 shows how, according to the present invention, the speaker selects in 
the text of the speech, words or terms for creating hyperlinl<s (I.e., associate 
URLs) to multimedia information or services. 

• Figure 6 shows, how accoitling to the present invention, the speaker creates a 
5 Speech Hyperlink Table that associates to each hyperlinked term, an address to 

access multimedia information or services. 

• Figure 7 shows how, according to the present invention, the speaker the speech 
recognition system trains before the speech. 

• Figure 8 shows how, according to the present invention, during the speech, the 
10 speech recognition system operating on the speaker workstation recognizes (i.e., 

word spots) the hyperlinked temis as they are pronounced by the speaker. 

. Figure 9 illustrates how. according to the present invention, when a hyperlinked 
term is recognized (word spotted) by the speech recognition system, the 
recognized term, the associated address and the universal time of the 
15 . recognition are stored as in a Speech Hyperlink-Time Table located on a Speech 
Sen/er connected to a communications network (e.g., to the Internet). 

. Figure 10 shows an example of a Speech Hyperlink-Time Table stored on a 
Speech Server according to the present invention. 

• Rgure 11 shows how, according to the present Invention, an auditor perceiving 
20 a term corresponding to an interesting topic during the course of the speech, 

selects this term simply by pressing a resen/ed key on a portable computer. 

• Figure 12 illustrates how. according to the present invention, the universal times 
at which interesting terms are selected by the auditor during the speech, are 
stored in a Selections Hyperlink-Time Table located in the auditor device 

15 workstat i on . 
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• Figure 13 shows how. according to the present invention, by connecting the 
auditor device to the communications network, the auditor updates the 
Selections Hyperlink-Tlme Table located in his workstation using the infomiatlon 
comprised in the Speech Hyperlink-Tlme Table stored on the Speech Server. 

• Figure 14 shows how. according to the present invention, hyperlinked terms are 
Identified and copied, jointly with the associated Uniform resource Locators 
(URLs), from the Speech Hyperlink-Tlme Table stored on the Speech Server to 
the Selections Hyperlink-Time Table on the auditor device. 

Figure 15 shows how. according to the present invention, an auditor selects from 
the updated Selections Hyperlink-Tlme Table an hyperiinked term (corresponding 
to an Interesting topic selected by the auditor during the speech), and activates 
tile associated hyperiink. 

Figure 16 shows how. according to the present Invention, multimedia infomiation 
or services hyperiinked to a selected term is accessed through the 
communications networic and retrieved on the auditor device. 

Figure 17 shows the steps of creating a Speech Hyperiink Table and of training 
a speech recognition system to identify hyperiinked terms, according to the 
present invention. 

Figure 18 shows the steps of creating a Speech Hyperlink-Time Table on a 
Speech Server and of recognizing hyperiinked terms pronounced during a 
speech, according to the present Invention. 

Figure 19 shows the steps of creating a Selections Hyperiink-Time Table on the 
auditor device and for selecting topics of interest during the course of a speech, 
according to the present invention. 
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Figure 20 shows the steps of retrieving Uniform Resource Locators (URLs) and 
for having access to the infomnation or services related to selected hyperlinked 
terms. 



Preferred embodiment of the invention 

5 As shown in Figures 1 and 2, the present invention discloses a system and a 

method for enabling a spectator (100) of a live event (e.g.. attending to a 
conference) (102) (202), or a radio auditor or television viewer (200) attending a live 
broadcast program, to select one or several topics (i01). (201) drawing his attention 
or interest, and to immediately or at a later time, to easily have access to multimedia 

10 infomiation related with the selected topics (103), (203). 

Figure 1 shows a typical situation according to the present Invention. An 
auditor (100) attending a live speech (e.g., a conference about the subject "Wine & 
Health"), has his attention drawn by a topic (101) (e.g.. the term "Resveratron for 
which he would like to have further infonnation. Under these circumstances, there is 
15 a need to provide a simple mechanism to enable an auditor (100) to select topics 
during the course of a speech (e.g., the terms Tannis", "Phenols", "Resveratrol" ). 
and to access immediately or at a later time, infomnation related to the selected 
topics on a server connected, for instance, to the Internet. 

Figure 2 illustrates another typical situation. A television viewer (200) attends 
20 a live broadcast speech (e.g., to a live television program about the subject "Wine & 
Healtif). 

As shown in Figure 3, hyperlinks (i.e.. URLs) (303) are associated with terms 
or words (304) likely to be cited by the speaker (301) in the course of his speech. 
These hyperlinked terms or words are related to relevant topics or items (302). A 
05 speech recognition system (305) operating on a speaker workstation (306) (i.e.. a 
imputing system a with a microphone (307) connected to it) detects during the 
speech, the pronunciation by the speaker of anyone of said hyperlinked terms (308), 



FR9-2002-0065 




and records the time (309) at which the detected hyperlinked term has been 
pronounced. The system is also based on the synchronization (312) of the speaker 
device (306) with several auditors devices (311) (e.g., workstations, portable 
computers, personal digital assistants - PDAs , smart phones, or any other type of 
5 hand held computing devices) according to a same universal time (309) (310), so 
that the flow of information transmitted by the speaker (301) and received by the 
auditors (300) is always synchronized, independently of the relative positions of 
speaker (301) and auditors (300). Each time an auditor (300) perceives an 
Interesting topic (313) during the speech, he immediately selects this topic simply by 
10 pressing a reserved key (314) on the auditor's device (311). Universal times (310) at 
which topics are selected (314) by the auditor (300) are stored on the auditor device 
(311). 

As shown in Figure 3, the. invention is based on the following principles : 

1. the synchronization - independently of the relative positions of the speaker 
15 (301 > and auditors (300) - of the speaker workstation (306) and auditors 

devices (311) according to a the same universal time (312) (e.g., the GPS 
time, as provided by GPS receivers (309), (310), and 

2. the detection by means of a speech recognition system (305) operating on 
the speaker workstation (306) (with a microphone connected to it) of 

20 hyperlinked terms (304) (e.g., ""Resveratror (308)) pronounced by the speaker 

(301 ) during the course of the speech (302). 

Universal Timing Systems 

Common timing sequences, independent of locations of speakers and 
25 auditors, can be derived from an absolute timing reference such as, for example, the 
Global Positioning System (GPS) time or the Universal Time Co-ordinated (UTC) 
time (also known today as GMT and ZULU time). 

To transmit precise timing signals, the GPS uses 24 satellites in SS** inclined 
orbits 10,000 miles above the earth. These timing signals are used by any GPS 



receiver anywhere on t^arth to determine Its position. A Wk MHz transmission 
carries a 1-MHz bandwidth phase-modulated signal named the clear acquisition 
(C/A) code. When a GPS receiver receives this signal from at least three GPS 
satellites, it can determine its own latitude and longitude with an accuracy of about 

5 30 meters. Apart the determination of geographical positions, the GPS is today 
widely used for distributing Precise Time and Time Interval (PTTI). The system uses 
time of anival (TOA) measurements for determining a position. A precisely timed 
clock is not essential for the user because time is obtained In addition to position by 
the measurement of the TOA of four satellites simultaneously In view. If the attitude 

10 over sea level is known, three satellites are sufficient. If the user Is stationary at a 
known position then, in principle, the time can be obtained by the observation of a 
single satellite. Information about the GPS time service is supplied by the 'Time 
Service Department", U.S. Naval Obsen/atory, Washington. DC. at 
http://tycho.usno.navy.mil/. 
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GPS Is today the world's principal supplier of accurate time. It is used 
extensively both as a source of time and as a means of transferring time from one 
location to another. Three kinds of time are. available from GPS: GPS time, UTC as 
estimated and produced by the Unrted States Naval Observatory, and the times from 
20 each free-running GPS satellite's atomic clock. The Master Control Station (MCS) at 
Falcon Air Force Base near Colorado Springs, Colorado, gathers the GPS satellites- 
data from five monitor stations around the worid. A Kalman filter software program 
estimates the time error, frequency error, frequency drift and Keplerian orbit 
parameters for each of the satellites and its operating clock. This infomnation Is 
25 uploaded to each satellite so that it can be broadcast in real time. This process 
provides GPS time consistency across the constellation to within a small number of 
nanoseconds and accurate position determination of the satellites to within a. few 
meters. 

30 The second universal time standard. Universal Time Co-ordinated (UTC), 

Introduces leap seconds to remain synchronized with the rotation of the earth. In 
order to provide an estimate of UTC time derivable from a GPS signal, a set of UTC 
corrections is also provided as part of the GPS broadcast signal. This broadcast 
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message includes the^e difference in wliole seconds between GPS time and 
UTC. This complicates software that deals with the smooth flow of data streams or 
calculates the times between data samples. GPS Time is preferred In this invention 
as this system avoids the introduction of leap seconds and Is easily related to UTC 
Infomiation about UTC (GMT) time service : can be found on: 
httpyAime.greenwich2000.com/. 



GPS receivers 



A Direct-to-Dlgital GPS Receiver is described in the following Web site : 
http://w3.research.ibm.com/present/gto200038.htm. 

10 This is an example of a tiny low cost chip which can integi^te GPS into anything, 
(e.g.. a PDA. a mobile phone, a wearable computer, a video camera) . This receiver 
has been jointly developed between IBM and Leica. The high speed analog 
capabilities of SiGe technology, when Integrated with the CMOS technology, allows 
the integration of this single chip directly to a digital . GPS (Global Positioning 

15 System) receiver. GPS derived position infomiation is finding a multitude of diverse 
applications: from mapping and surveying to vehicle tracking to 91 1 cell phone caller 
location to automated fami equipment to even robotics golf carts... This receiver chip 
reduces the radio dimensions and complexfty. There are no analog mixer stages 
nor costly discrete components, (such as high quality filters) that conventional two 

20 stage analog down conversion would have required. Instead, the incoming GPS 
signal is literally digitized right at the antenna, then filtered digitally in a CMOS based 
chip. This direct digitization is made possible by the ability of SiGe technology to run 
at high speed on very little power, and the core of this technology Is a SiGe based 
Analog to Digital Data Converter. 

25 According to the present invention GPS or GLONASS receivers must be 

integrated or connected to the speaker workstation (typically a Personal Computer) 
and to auditoi^s devices (e.g.. Personal Computers, wearable computers. Personal 
Digital Assistants (PDAs), smart phones or onboard mobile, computers). The 
universal timing signals that are received from GPS or GLONASS satellites, are 

30 used to initialize and synchronize the internal electronic clocking systems on the 
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speaker workstation a^auditor's devices according to th^tme universal time. 
During the periods on wliich GPS or GLONASS satellites are out of sight (e.g.. when 
speaker or auditor's devices are inside buildings or not connected to an external 
antenna), and no timing signals are thus received from those satellites, timing 
5 Information must be continuously derived from the autonomous electronic clocking 
systems. Depending on the drift of the Internal clocking systems, and to keep 
enough timing precision and to be sure that user's devices are synchronized with 
speaker's workstations and broadcasting stations at the same universal time, a more 
or less frequent periodic reception of satellite signals must be perfomned. In practice. 

10 

• if the user device is a portable or in-vehicle mounted device, satellite signals will 
be received when the user is out of doors or is traveling. 

• If the user device is fixed or installed in a house or building during long periods, 
the user device must be connected to an outdoors installed GPS or GLONASS 

15 antenna, (e.g., antenna Installed on the roof of the building). 



Systems and Methods for Detecting Words in a Continuous Speech 

Speech recognition is the process by which a computer (or other type of 
system or apparatus) Identifies spoken words. Basically, a speech recognition 
system is a computer which can correctly recognize what a speaker is saying. 

20 The speech recognition is an extremely difficult task. Unlike written text, no 

clear spacing exists between spoken words. Full sentences or sentences are 
typically pronounced without any pause. Furthermore, the acoustic variability in the 
speech signal typically precludes, an unambiguous mapping to a sequence of words 
or subword units, such as pronunciations of consonants and vowels. A major source 

25 of variability In speech is coarticulation. or the tendency for the acoustic 
characteristics of a given speech sound or phone sound to differ depending upon 
the Dhonetic context In which it is produced. 



19 



Speech recogn^ systems can be categorized depending on the speaking 
styles, the vocabularies, and language models they accommodate. Isolated word 
recognizers require speakers to Insert brief pauses between individual words 
Continuous speech recognizers operate on fluent speech, bgt typically employ strict 
5 language models, or grammars, to limit the number of allowable word sequences. 

Word spotters are a particular kind of speech recognizers. They operate also 
on fluent speech. However, rather than providing a full ti^nscription. word spotters 
selectively locate relevant words or sentences. Word spotting is useful to. retrieve 
infomiatlon based on keyword indexing or to recognize isolated words in voice 
10 command applications. 

Multiple speech recognition systems are today available with the word 
spotting capability required to support the present invention. These systems allow 
the detection of predefined words or sentences (hyperiinked terms) that may. be 
pronounced by a speaker in the course or a speech. Such speech recognition 
15 systems are, for instance, described In the following patents: 

US patent 5.199.077 entitled "Wordspotting for voice editing and indexing" 
discloses a technique for wordspotting based on hidden Mari<ov models (HMM's)' 
The technique allows a speaker to specify keywords dynamically and to train the 
associated HMM's via a single repetition of a keyword. Non-keyword speech is 
20 modeled using an HMM trained from a prerecorded sample of continuous speech 
The wordspotter Is intended for interactive applications, such as the editing of voice 
mail or mixed-media documents, and for keyword indexing in single-speaker audio 
or video recordings. 

US patent 5.425,129. entitled "Method for word spotting in continuous 
25 speech", discloses a system and method for analyzing a digitized speech data 
channel for the presence of words or sentences from a desired list. The system and 
method of this Invention, that has been Implemented in the context of the IBM 
Continuous Speech Recognition System (ICSRS). provides In a speech recognition 
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system a subsystem for spotting those pre-specified words or sentences In the 
presence of unrelated phonetic data. 

US patent 5,950,159 entitled "Word spotting using both filler and phone 
recognition" discloses a word-spotting system and a method for finding a keyword in 

5 acoustic data. The method includes a filler recognition phase and a keyword 
recognition phase wherein: during the filler recognition phase the acoustic data Is 
processed to identify phones and to generate temporal delimiters and likelihood 
scores for the phones; during the keyword recognition phase, the acoustic data is 
processed to identify instances of a specified keyword Including a sequence of 

10 phones; wherein the temporal delimiters and likelihood scores generated in the filler 
recognition phase are used in the keyword recognition phase. 

US. patent 6,006,185 entitled "System and device for advanced voice 
recognition word spotting" discloses a speaker independent, continuous speech, 
word spotting voice recognition system and method. The edges of phonemes In an 
15 utterance are quickly and accurately Isolated. The utterance is broken Into wave 
segments based upon the edges of the phonemes. A voice recognition engine is 
consulted multiple times for several wave segments and the results are analyzed to 
correctly identify the words In the utterance. 

US patent 6.073.095 entitled "Fast vocabulary independent method and 
20 apparatus for spotting words in speech" discloses a fast, vocabulary independent 
method, for spotting words in speech by utilizing a preprocessing step and a 
coarse-to-detailed search strategy for spotting a word/phone sequence in speech. 

US patent 6,185,527 entitled "System and method for automatic audio 
content analysis for word spotting, Indexing, classification and retrieval" discloses a 
25 system and method for indexing an audio stream for subsequent information 
retrieval. 



US patent 6,230,126 entitled "Word-spotting speech recognition device and 
system" discloses a device for speech recognition which includes a dictionary which 
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stores features of reco^lon objects. The device further Sdes a matching unit 
which compares features of Input speech with the features of the recognition 
objects, and a dictionary updating unit which updates time lengths of phonemics in 
the dictionary based on the Input speech when the matching unit finds substantial 
5 similarities between the input speech and one of the recognition- objects. 

A fundamental problem with speech recognition systems is the unreliable 
behavior of the state-of-the-art technology for performing "word spotting" (I e 
Identification of pre-specifled keywords or temis) in a continuous mode' 
independently of the speaker, based on unknown or generic speaking styles' 
10 vocabularies, noise levels and language models. On the contrary, the automatic 
word spotting function required for the present Invention is relatively easy to 
implement since the system can be trained and tuned In advance by the speaker 
using his own voice, his specific speaking style and .a specifically adapted 
vocabulary and language model. 
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According to one aspect of the Invention, a system and a method Is disclosed 
for creating hyperlinks from pre-selected utterances (304)- (commonly spoken words 
terms or sentences) intended to be pronounced by a speaker (301) during a speech 
(302). to related relevant data (303) on the Web. and for training a speech 
recognition system (305) to identify (word spot) said hyperllnked terms (304) 
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According to another aspect of the invention, a system and a. method is 
disclosed for automatically recognizing (word spotting) the hyperllnked terms (304) 
as they are pronounced by a speaker (301) during a live speech, and for creating on 
a network server a table comprising a list of recognized hyperllnked terms (304) the 
associated network addresses (i.e.. the URLs) (303) and the universal times (309) 
25 corresponding to the recognition of these hyperllnked temis. 

According to a further aspect of the Invention, a system and a method is 
disclosed for recording on an auditor device (311) universal times (310) 
corresponding to a selection (314) by an auditor (300) during the course of a speech 
(302). of one or several Interesting topics (313). 
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According to another aspect of the invention, a system and a method Is 
disclosed for enabling an auditor (300) to access and retrieve from a server 
connected to a computer network, relevant infomriatlon related to an hyperlinked 
term (304). 

5 Method for creating a Speech Hyperlink Table and for training a Speech 
Recognition System with the vocabulary of hyperlinked terms 

As shown in Figure 17, the invention discloses a system, method and 
computer program for use in by a speaker (301) before a speech, for creating 
hyperlinks from selected utterances (304) (commonly spoken words or sentences) 
10 intended to be pronounced by a speaker (301) during a speech (302). to related 
relevant data (303) on one or several senders connected to a computer network 
preferably the Web, and for training a speech recognition system (305) for word 
spotting the speech with the vocabulary of said hyperlinked temis (304). The method 
comprises the steps of: 

15 during the production or edition of the speech: 

• (1 701 ) editing the text or a draft text (400) of the speech; 

. (1702) selecting and marking on the speech text (500) a plurality of relevant 
temis. or words (501 ) for which hyperlinks must be created; 

• (1703) creating a Speech Hyperlink Table (600) for the speech; 

20 • defining hyperlinks In the Speech Hyperlink Table (600) between the selected 
temis or words (501) and multimedia information or services located on one or a 
plurality of senders (909) connected to a computer network (908). said step 
comprising the further steps of: 

for each one of the selected hyperlinked temns or words (501): 

25 • (1704) assigning a name and/or a description (601) (preferably a brief 
description); 
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• (1705) assigning a destination address (602) (for instance the URL) within the 
network (908) to access the desired multimedia Infomiation or service; 

• (1706) storing in the Speech Hyperlink Table (600) the assigned name (or 
description) (601 ) and/or the destination address (602); and 

5 once hyperlinks have been defined in the Speech Hyperlink Table (600): 

• (1707) training a speech recognition system (701) running on the speaker 
workstation (702) with the vocabulary of the hyperiinked terms (602), (703) for 
automatically recognizing these hyperiinked terms when they will be pronounced 
by the speaker (700) during the speech. 

10 Figure 4 shows the text (400) of an exemplary speech (e.g., a conference 

about the subject "Wine & Healtff), to be perfomn In a conference room or to be 
broadcast at the radio or television. During the live speech, this text can be read in 
its entirety by the speaker (301) or the speaker can intentionally omit to read of 
some parts, change the order, or introduce alternative or additional comments at 

15 convenience. 

Figure 5 illustrates how a speaker, a program editor (or any person committed 
do to it) takes the speech text (500) and selects certain terms (501) (words or 
sentences, such as "Phenols", "Resveratrol", "Tannins", "Eliagic acid", 
"Hydroxycinnamates", "Free radicals", "Anttiocyanin", "Gallic acid" ) to associate 
20 additional Information accessible on the Web. Each of these selected terms named 
hyperiinked terms, must be associated to a network address (I.e., to the 
con-esponding URL) on the Web from where the related information or service can 
be retrieved. 

Figure 6 shows how the speaker, the program editor (or any person 
25 committed do to it) creates on the speaker workstation (306) a Speech Hyperiink 
Table (600) that associates with each selected hyperiinked term (601) (I.e., selected 
word or sentence on the speech text, e.g., "Resveratrol"), a corresponding URL on. 
the Web (602) (e.g., http://www.ag.uiuc.edu/~fffVresvera.litml) 
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Figure 7 shows how. before the speech, the speaker (700) trains with the 
vocabulary of hyperlinl<eci terms (703) (e.g., "Phenols", "Resveratrol", Tannis", ... ) a 
speech recognition system (701) with word spotting capability installed on his 
workstation (702). In a particular embodiment of the invention, the word spotting 
5 function is implemented by means of the IBM Continuous Speech Recognition 
System - ICSRS - running on the IBM ViaVoice software product. 

Method for recognizing hyperlinked terms during a speech and for creating a 
Speech Hyperlink-Time table on a Speech Server 

As shown in Figure 18, the invention also discloses a system, method and 
10 computer program for use in a speaker workstation (802) for recognizing 
h^n^erlinked terms (803) as they are pronounced by the speaker (800) during a live 
speech, for creating on a Speech Server (907) connected to a network (908), a 
Speech Hyperlink-Time Table (906), and for updating this table with records 
comprising the sequence (905) of the recognized hyperlinked terms (304), the 
15 corresponding network addresses (i.e., the URLs) (303) and the universal times 
(309) when these hyperlinked terms have been recognized. More particularly, the 
method comprises the steps of: 

• (1801) creating a Speech Hyperlink-Time Table (906) for the speech on a 
Speech Server (907) connected to a network (908); and 

zO during the speech: 

• (1802) "word spotting" the speech to recognize the pronunciation by the speaker 
(800) of hyperlinked terms (803) by means of a speech recognition system (801) 
running on the speaker workstation (802) and purposely trained for the speech; 
and; 

'jr for each hyperlinked term (803) that has been recognized: 
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• (1803) determining by means of an universal time device, the universal time 
(904) con-espondlng to the recognition of the hyperllnked term (903); 

• (1804) creating a record Including: 

• the universal time (904) when the hyperllnked term (903) has been 
5 recognized; 

• the name or short description (601) of the recognized hyperllnked term 
(803) using the Speech Hyperlink Table (600); and 

• th© network address (1005) (I.e., the URL) corresponding to the 
recognized hyperllnked temi (803) using the Speech Hyperlink Table 

10 (600); 

• (1805) storing the record in the Speech Hyperllnk-Tlme Table (906), (1000) 
stored on the Speech Server (907) accessible through the network (9.08) from 
the speaker workstation (902). 

Figure 8 illustrates how, while the speaker (800) Is pronouncing his speech. . 
15 the "word spotting" function running on the speech recognition system (801) installed 
on the speaker . workstation (802) (e.g., Implemented by means of the /BM 
Continuous Speech Recognition System - iCSRS - running on the IBM VlaVolce 
software product) and previously trained by the speaker^ automatically detects the 
hyperllnked temis (803) from the voice stream. 

20 Figure 9 shows how. after recognition of the hyperllnked term (903) (e.g.. 

"Resveratrol") by means of the speech recognition system (901) running on the 
speaker workstation (902), 

• the universal time (904) (e.g., 12/05/2001 14:23:18) corresponding to the 
recognition of the hyperllnked term (903); 

25 • the hyperllnked term name or short description (601) (e.g., "Resveratrol'); and 

• the associated URL (602) (e.g., http://www.agMiuc.edu/~m/resvera.litml); 

(last both extracted from the Speech Hyperlink Table (600) are sent (905) and 
stored on a Speech Hyperlink-Time Table (906) located on a Speech. Server 
(907). 
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Figure 10 shows an example of a Speech Hyperlink-Time Table (1000) (906) 
stored on the Speech Server (907) as It appears at the end of the speech (e.g., 
about the subject "Wine & Health" {^ 001)). The header of this table comprises the 
URL (1002): http:/Amw.directbuyer.com/conference-0173.htm/ or netwoi1< address 

5 where the Speech Hyperlink-Time Table (1000) has been stored in the Speech 
Server (907) {e.g., www.directbuyer.com). The URL where the Speech 
Hyperlink-Time Table (1000) can be found in the network, must be known in 
advance by the auditors. Each row in this table corresponds to a hyperllnked temn 
(903) pronounced by the speaker and recognized by the speech recognition system 

10 (901) during the speech. Each column corresponds, respectively to: 

• (1003) the universal time corresponding to the recognition of a spoken 
hyperllnked temn (903) by the Speech Recognition System (901) during the 
speech; 

• (1004) the name and/or short description (601) of the recognized hyperllnked 
15 (903) temri copied from the Speech Hyperlink Table (600); 

• (1005) the URL (602) corresponding to the recognized hyperllnked terni 
(903), copied from the Speech Hyperlink Table (600). 

Method for creating a Selections Hyperllnk-Time Table and for selecting topics 
of Interest 

20 

As illustrated by Figure 19, the invention discloses also a system, method and 
computer program for creating a Selections Hyperlink-Time Table (1106) on an 
auditor device (1102) and for recording in this table a sequence of universal times 
(1105) corresponding to the selection (1104) by an auditor (1100) of interesting 
25 topics (1 103) during the speech, for the purpose of. immediately after the speech or 
at a later time, receiving further information related to these topics. More particularly, 
the method for use in the auditor device, comprises the steps of: 



• (1901) creating a Selections Hyperlink-Time Table (1200) on the auditor device 
(1102); 
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• (1902) recording on the Selections Hyperlink-Time Table (1200) the network 
address (i.e., the URL) (1201) of the Speech Hyperlink-Time Table (906) stored 
on the Speech Server (907); 

• (1 903) listening to the speech (1 1 07) pronounced by the speaker (1 1 01 ); 

5 • (1904) perceiving In the speech, a topic of interest (1103) for which additional 
information or service is desired; 

• (1905) selecting this topic of interest (1103) by entering a: selection command 
(1 1 04) on the auditor device (1 1 02); 

• (1906) determining the current universal time (1105) by means of an universal 
10 time device (for instance a GPS receiver) integrated or connected to the auditor 

device; . . 

• (1907) recording this current universal time (1105) in the Selections 
Hyperlink-Time Table (1106), (1202). . . 

Figure 1 1 illustrates how during the speech, the auditor (11 00) operates using 
15 a portable auditor device (1 102) in stand-alone mode (i.e., isolated, not connected to 
a network). According to the particular embodiment detailed In Figure 11, each time 
the auditor (1100) perceives during the speech an interesting topic (1103), he 
Immediately selects this topic simply by pressing a reserved key (1104) on the. 
auditor device (1102). Universal times (1105) con-esponding to the selection of the 
20 topics by the auditor, are stored in the Selections Hyperlink-Time Table (1106) on 
the auditor device (1 1 02). 



Figure 12 shows a typical Selections Hyperlink-Time Table (1200) created on 
the auditor device (1 102) during the speech. Each row in this table corresponds to a. 
different universal time corresponding to the selection by the auditor of an Interesting 

25 topic during the speech. The header of this table comprises the URL (1201) (e.g.,. 
hnp://www.dlrectbuyer.com/conference-0173.htm/ ) of the Speech Hyperilnk-Tlme 
Table (906),(1000) con-esponding to the speech. The Speech Hyperiink-Tlme Table 
(906),(1000) on the Speech Server (907) {e.g., www.directbuyer.com). Is updated 
during the speech from the speaker woricstation (902). As It has been said before, 

30 the URL of the Speech Server must be provided in advance to the auditors to allow 
them to create locally a Selections Hyperiink-Time Table (1200). 
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Method for retrieving hyperlinlcs and for accessing information 

As Illustrated by Figure 20, the invention discloses also a system, method and 
computer program for enabling an auditor (300) to access and retrieve from the 
Web relevant data or infomriation related with the hyperlinked terms (304) that were 
5 active when the auditor made the corresponding selections (314). More particularly, 
the method for use on the auditor device (1300), comprises the steps of: 

• (2001 ) accessing the Speech Server (1 303); 

for each universal time (1202) recorded in the Selections Hyperlink-Tlme Table 
(1304), (1200) located on the auditor device (1300): 

10 • (2002) sending (1306) to the Speech Server (1303) the recorded universal 
time; 

• (2003) identifying in the Speech Hyperlink-Time Table (1302), (1000) located 
on the Speech Server (1303), the hyperlinked term (1004) associated with 
said universal time (1306) (hyperlinked term that was active at that universal 

15 time); 

• (2004) retrieving (1307) from the Speech Hyperlink-Time Table (1302), (1000) 
located in the Speech Server (1303) the name (or description) (1004) and the 
destination address (URL) (1005) of the selected hyperlinked term; 

• (2005) storing the retrieved hyperlink name and destination address (URL) 
20 (1307) into the Selections Hyperlink-Time Table (1304), (1402) located on the 

auditor device (1300); 

The method includes the further steps of: 

• (2006) selecting in the Selections Hyperlink-Time Table (1502), an hyperlink 
25 (1501) using the hyperlink name (or description) or the associated destination 

address retrieved from the Speech Server (1507); 
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• (2007) activating said hyperlink (1501) using a browser program running on the 
auditor device (1500); 

• (2008) accessing the Infomnation and/or service located on a server (1506) 
connected to the network (1505) using the retrieved destination address (1503) 

5 (1504) associated with the selected hyperlink (1501); 

• (2009) retrieving (1 601 ) the Information and/or service (1 604) from the accessed 
sender (1603) on the network (1602); 

• displaying, on the auditor device (1600). the retrieved Information and/or service 
(1604) using a browser program 

10 Figure 13 illustrates how, by connecting the portable device (1300) of the 

auditor (11 00) to the communications network (1301) (e.g., the Internet network) and 
by accessing the Speech Hyperlink-Time Table (1302) on the Speech Server (1303). 
the infomiation on the Selections Hyperlink-Time Table (1304) of the audrtor device 
(1300) is updated. 

15 

Figure 14 Illustrates In detail the process for updating the Selections 
Hyperllnk-Tlme Table (1400) on the auditor device (1300) using the Speech 
Hyperiink-Tlme Table (1401). (1302) on the Speech Server (1303). Basically, by 
means of this process, the hyperlinked terms (or short names and/or short 
2b descriptions) and the URLs of the hyperlinked terms that were active at the 
universal times (1403) when the auditor has selected Interesting topics, are identified 
in the Speech Hyperlink-Time Table (1401) located on the Speech Server (1303). 
and are copied into the Selections Hyperlink-Time Table (1402) located on the 
auditor device (1300). 



Figure 15 illustrates how the auditor selects and get access from the auditor 
device (1500) to Web pages hyperlinked to the topics he has selected during the 
speech. Basically, the auditor points to and selects a hyperlinked term (1501) on the 
updated Selections Hyperlink-Time Table (1502) and. by means of a software on the 
auditor device (1500), activates a Web browser and triggers the hyperlink to the URL 
(1503) of the selected item (1501). In the example shown In this figure, the auditor 
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selects the hyperlinke^term "Ellagic acid", (in fact a chem^^compound found in 
wines), and triggers the hyperlink pointing to the URL (1504): 
ht^://www.hopeforcancer.com/ellagicacid.htm 

Figure 16 shows how the Web page (1601), (e.g., document: ellagicacid.htm), 
5 associated with the selected hyperlinked term (1501), (e.g., "Ellagic acid"), is 
received through the network (1602) from the accessed Web server (1603) (e.g., 
http://www.hopeforcancer.com), and displayed (1604) or played on the auditor 
device (1600). 

While the invention has been particularly shown and described with reference 
10 to a preferred embodiment, it will be understood that various changes in form and 
detail may be made therein without departing from the spirit, and scope of the 
invention. 
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Claims 

What is claimed is : 

1. A method, for generating from a speal<er device, a Speech Hyperlinl<-Time table 
during a speech, said Speech Hyperlink-Time table being accessible by one or a 
5 plurality of auditors, comprising the steps of: 

• locating and accessing a Speech Hyperiink table comprising : 

• means for identifying one or a plurality of predefined hyperiinked terms 
intended to be pronounced by a speaker; 

• means for locating and accessing information related to each of said, one or 
10 plurality of predefined hyperiinked temis; 

♦ retrieving means for locating and accessing a Speech Hyperiink-Tlme table; . . 



during the speech : 



• recognizing, when they are pronounced by the speaker, the hyperiinked terms 
15 predefined in the Speech Hyperiink table by means of a. speech recognition 

system connected to the speaker device; , . 

for each recognized hyperiinked term : 

• determining an universal-time corresponding to the pronunciation by the speaker 
of the recognized hyperiinked term; 

20 • creating a new record on the Speech Hyperlink-Time table; 

• copying into said new record : 

• the unlversal-tlme corresponding pronunciation by the speaker of the 
recognized hyperiinked term; 

• means, retrieved from the Speech Hyperiink table, for identifying the 
25 recognized hyperiinked term; 

• means, retrieved from the Speech Hyperiink table, for locating and accessing 
information related to the recognized hyperiinked terni. 



FR9-2002-0065 ' * 1 



32 




2. The method according to the preceding claim comprising the preliminary steps of : 

• creating a Speech Hyperlink table, preferably in the speaker device; 

• defining, in said Speech Hyperlink table, hyperlinks to one or a plurality of 
5 servers where information related to predefined terms can be accessed. 

3. The method according to the preceding claim wherein the step of defining, in said 
Speech Hyperlink table, hyperlinks to one or a plurality of servers where information 
related to predefined hyperlinked terms can be accessed, comprises the further 
steps of: 

10 • predefining hyperlinked terms intended to be pronounced by the speaker during 
a speech; 

and, for each new predefined hyperlinked term : 

• creating a new record in the Speech Hyperlink table; 

• assigning and copying means for identifying the predefined hyperlinked term in 
15 said record; 

• assigning and copying means for locating and accessing information related to 
the predefined hyperlinked term in said record. 

4. The method according to any one of claims 2 to 3 wherein the step of creating a 
Speech Hyperlink table, preferably in the speaker device, comprises the further 

20 steps of: 

• assigning to said Speech Hyperlink table means for locating and accessing the 
Speech Hyperlink-Time table; 

• copying said locating and accessing means in the Speech Hyperlink table, 
preferably in a header. 
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5. The method according to any one of the preceding claims wherein the step of 
recognizing, when they are pronounced by the speaker, the hyperlinl<ed terms 
predefined In the Speech Hyperlink table by means of a speech recognition system 
connected to the speaker device, comprises the preliminary step, before the speech; 
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• training the speech recognition system to recognize the predefined hyperlinked 
terms when they are pronounced by the speaker. 

6. The method according to any one of the preceding claims wherein said speech Is 
a television or radio broadcast program. 

10 7. The method according to any one of the preceding clalrris wherein : 

• said Infomriation related to a predefined hyperlinked temi is located on one or a 
plurality of servers; 

• said means for Identifying a predefined hyperlinked term comprises a name 
and/or a description; 

15 • said means for locating and accessing information related to a predefined 
hyperiinked term comprises a destination address within a networi<; 

• said Speech Hyperlink-Time table Is located on a Speech Server; 

• said means for locating and accessing the Speech . Hyperiink-Tlme table 
comprises a destination address within a network. 

20 8. The method according to any one of the preceding claims wherein ; 

• said network Is an internet network; 

• said destination addresses are Unifonn Resource Locators; 

• information related to a predefined temn is a web page; 

• said servers are web servers. 

25 9. The method according to any one of the preceding claims comprising the step of: 
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• sending to one or a plurality of auditor devices means for locating and accessing 
the Speech Hyperlink-Time table. 

10. A speaker device comprising means adapted for carrying out the method 
according to any one of the preceding claims. 

5 11. A computer program comprising instructions for carrying out the method 
according to any one of claims 1 to 8 when executed on the speaker device 
according to the preceding claim. 

12. A method for selecting from an auditor device, one or a plurality of hypierlinked 
terms pronounced by a speaker during the course of a speech, and for accessing, at 
10 a later time. Information related to each of said one or plurality of selected 
hyperiinked terms, comprising the steps of: 

• each time a selection command Is received for selecting a term currently 
pronounced by a speaker during the course of a speech: 

• determining the current universal-time; 

15 • recording the current universal-time in a Selections Hyperiink-Time table; 

• accessing a Speech Hyperlink-Time table, said Speech Hyperiink-Tlme table, 
accessible by one or a plurality of auditor devices, comprising for each one of a 
plurality of predefined hyperiinked terms pronounced by the speaker : 

• an universal-time interval corresponding to the pronunciation of the 
20 predefined hyperiinked term; 

• means for identifying the predefined hyperiinked tenm; 

• means for locating and accessing infomnation related to the predefined 
hyperiinked term; 

for each universal-time recorded in the Selections Hyperiink-Time table: 

25 • identifying in the Speech Hyperiink-Tlme table, the selected hyperiinked term 
pronounced at the recorded universal-time; 
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• retrieving from the Speech Hypertink-Time table, said means for Identifying, and 
for locating and accessing the selected hyperlinked term; 

• storing the retrieved means for identifying the selected hyperilnked term and for 
locating and accessing infonnation related to the selected hyperilnked term into 

5 the Selections Hyperlink-Time table. 

13. The method according to the preceding claim comprising the preliminary step of: 

• creating the Selections Hyperiink-Time table for the speech. 

14. The method according to any one of claims 12 to 13 comprising the preliminary 
step of: 

10 • receiving means for locating and accessing a Speech Hyperlink-Time table. 

1 5. The method according to any one of claims 1 2 to 1 4 wherein : 

• infonriation related to a hyperlinked term is located on one or a plurality of 
servers; 

• said means for identifying a hyperilnked term comprises a name and/or a 
15 description; 

• said means for locating and accessing information related. to a hyperilnked term 
comprises a destination address within a network; 

• said Speech Hyperlink-Time table is located on a Speech Sender (803); 

• said means for locating and accessing the Speech Hyperlink-Time Table 
20 comprises a destination address within a network. 

1 6. The method according to any one of claims 1 2 to 1 5 wherein : 

• said network is an internet networi<; 

• said destination addresses are Uniform Resource Locators; 

• information related to a term is a web page; 
25 • said auditor device comprises a web browser; 
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• said servers are web servers. 

17. The method according to any one of claims 12 to 16 comprising the further step 
of: 

• receiving a display command for displaying Infomiation related to a hyperlinked 
5 term, said display command comprising means for identifying said hyperlinked 

term; 

• identifying, in the Selections Hyperlink-Time table, means for locating and 
accessing information related to said hyperlinked term; 

• retrieving and displaying said information. 

10 18. The method according to any one of claims 12 to 17 wherein said speech is a 
television or radio broadcast program. . 

19. An auditor device comprising means adapted for carrying out the method 
according to any one of claims 1 2 to 1 8. 

20. A computer program comprisirig instructions for carrying out the method 
15 according to any one of claims 12 to 18 when executed on the auditor device 

according to the preceding claim. 
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SYSTEM AND METHOD FOR ENHANCING LIVE SPEECH WITH 
INFORMATION ON THE WORLD WIDE WEB 

Abstract 

The present Invention Is directed to a system, method and computer program 
5 for enabling an auditor of a live speech to access Immediately after or at a later time, 
complementary Infomiatlon related with temis pronounced during this speech. The 
system associates hyperlinks (I.e.. URLs) with selected tenns or words likely to be 
pronounced by the speaker in the course of the speech. A speech recognition 
system operating on a speaker device (i.e., a computing system a with a microphone 

10 connected to It) recognizes during the speech (i.e., word spotting) the pronunciation 
by the speaker of anyone of said hyperlinked terms, and records the time at which 
each recognized hyperlinked term has been pronounced. The system Is also based 
on the synchronization of the speaker device with several auditors, devices (e.g., 
workstations, portable computers, personal digital assistants - PDAs . smart phones, 

15 or any other type of handheld computing devices) according to a same universal 
time, so that the flow of information transmitted by the speaker and received. by the 
auditors is always synchronized. Independently of the relative positions of the 
speaker and auditors. Each time the auditor perceives an Interesting topic during the 
speech, he Immediately selects the topic simply by pressing a resen/ed key on the • 

20 auditor's device. Universal times at which topics are selected by the auditor are 
stored In the auditor device. 



Figure 3 
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Fig. 1 : During a speech auditors perceive interesting topics 
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Fig. 2: During a radio or television program an auditor perceives 
interesting topics 
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(304) 



(303) 





URL r 


-Phenols" / 


http:/AvvvwstonoWlhvlneiy.com/h8aim,html / 


-Resveratror / 


httpyAyww.ag.uluc.edu/~ffh/resvera.h^ ^ 


Tannins" 




"Ellaglcacfcf 


hnp:/Avviw.hcveforcanc8rxom/enagrcacldJ>tm 


"Hydroxycinnamates" 


hapyA¥vw.svi«t8.nV'sps/joumals/l?b360008^ 


"Ravon-a-ols" 


http:/Amv.fiya3.ofg/scItech/8urafconL01 042ahtml 


Trea radicals'* 


mtp:/A«ww.vvho^|.eo/)jubltoattona^ 


"Anthocyarin* 


http://Wvw.orstediiWept^yss01/!Emt^^ 


"Xaafficadd" 


http/^AnAntf.vtn3etaaiite.comABa5eJitrnt 




(308) 

.7\ 



\esverdtrol 
loses gRuosfof 
its molecule ... 




(302) 



Speech Recognition System 
(305) 

Spealcer 
workstation 
(306) 

(309 



Synclironized universal time 
(312) 



Speaker 
(301) 



Resveratrol 

•7 



(310) 

Auditor device 
(311) 





(313) 



Auditor 
(300) 



Principles of the 
Invention (314) 

Fig. 3: The spealcer worlcstation and auditors devices are 
synchronized. Hyperlinlced terms pronounced by the 
speaker are automatically recognized during the speech 
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(400) 



The concentration and molecular structure of the phenols changes as ^ 
grapes are processed into wine, and as new wine ages into old wine. i 
Oxidation (the addition of an oxygen atom to a molecule) taices place, 
and the resulting molecules further interact with each other. 
Resveratrol loses glucose of its molecule. The oak barrel tannins 
break down and release eilagic acid. After a few months, the ^ 
hydroxycinnamates begin to break down. The "flavon-3-ols from ;• 
bonds with other molecules and change into a form that is not found 
in the fresh grape. | 



The good news about all these chemical changes is that as wine ages, :^ 
the number of anti-oxidlzing molecules in it actually increases. 
Anti-oxidants purge the body of free radicals and are good for us. • 

i 

The bad news is that, despite the Increase in the number of 
anti-oxidizing molecules over time, their bioavailability decreases. 
The anti-oxidizing molecules that are formed during aging are much | 
larger than the original phenols, and thus are not well absorbed in the I 
alimentary canal. After one or two years of aging, the amount of 
anthocyanin has decreased to about one-sixth of the original level. 
After five to ten years, wine has lost most of the smaller, and | 
presumably health friendly, phenols. "When we look at very old wine, | 
the only component that stands out is gallic acid". 

So, ironically, a $7 bottle of 3-year old wine is a healthier drink than a 
$600 bottle of 25-year old wine. (Of course, it may not taste as good!). 



Aoitd Wim Loses Its Vitality, hy: Atidrev; L Waterhousc, PhD, Dep.ijf Viticulture avi Btology at the University of Caljfimna at Davis 
Conffraxce on Alcohol and Wine in Health and Disease, New York Academy of Sciences, April 26^29, 2001 



k 



Kryas.org/scitech/sMm/conJJ} l_0^26.html# 13 



Fig. 4: The speaker prepares the speech text 
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(5Q1) 



The concentration and mo 
grapes are processe 
Oxidation (the a^pldiffon 
and the re 

Resveraifroi loses gliicose 
brealc down ancM'elease 



elfie 



^ _ jom t\a 

tiles fur6i4r interact 

oleciile. The 
eiiadic^cid. Mier a few 



(500) 

^ -7^ 

>fe^the phenols changes as 
ages into old wine. 
G^jle) taices place, 
each other, 
k barrel t^nins 
lonths, the 



hydroxYcinnafnates begin tb break do^Vi. The f lavdn-3-Qls from 
bonds with other molecuf^s and chanj i into a form that is not found 
in the fresh grape. 



The good news about all these chemical 
the number of anti-pkidizing molecules 
Anti-oxidants purge the body of free 



llchanges is that as wine ages, 
in it actually increases. 
radicals and are good for us. ' 



The bad news i^that, despite the increas^ in the number of 
antl-oxidizing/iolecules over time, their bioavailability decreases. 
The antl-oxidiiing molecules that are forced during aging are much 
larger than the original phenols, and thus kre not well absorbed in the 
alimentarWbanal. After one or two years ofl aging, the amount of 
anthocya^in has decreased to about one-sbxth of the original level. 
After fiv6 to ten years, wine has lost most clf the smaller, and 
presurhably health friendly, phenols. " Whei we look at very old wine 
the only component that stands put is galli6 acid ". ' 

So, ironically, a $7 bottle of 3-year old wine is a healthier drink than a 
$600 bottle of 25-year old wine. (Of course, k may not taste as good!). 



1 



ConJ^m on Alcohol and Wine in HcaUh and Disease, New York Academy of Sciences. April 26-29, 2002 
hup:/ / www.,tyas.otg/scilech/sHm/conJ_Ol_0426.htmtf^l3 

Fig. 5: The speaker selects hyperlinked terms in the text 
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Speech Hyperlink Table (^po) 



Hyperlinked 

loi 1 1 1 


UKL 


"Phenols" 


http://www.stonehillwinery.conri/health.html 


"Resveratrol" . 


http://www.ag.uiuc.edu/~ffh/resvera.html 


"Tannins" / 


http://www.mangiarebene.net/wine/manual/health.htnfil 


"Ellagic acid/ 


http://www.hQpeforcancer.com/ellagicacid.htm 


"Hydroxyci^namates" 


http://www.swets.nl/sps/journals/pb36000s.html 


"Flavon-3^-ols" 


http://www/nyas.org/scitech/sum/conf _01_0426.html 


"Free radicals" 


http://w\Afl^.who.sci.eg/publications/emhj/0402/21 . htm 


"Anthdcyanin" 


http://w;ww.orst.edu/dept/Ipi/ss01/anthocyanin.html 


"Gal/c acid" 


http://www.vinsetsante.com/za5e.html 


(601) 


(602) 



Fig. 6: The speaker creates a Speech Hyperlink Table and 
associates an address with each hyperlinked term 
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(703) 



Vocabulary of 
hyperlinked terms 



"Phenols" 



"Resveratrol" 



"Tannins' 



(j'ResveratroV'\ 



Speaker 
(700) 




Speech Recognition System 
(701) 



Ffg. 7: Thie speaker trains the Speech Recognition 
System to recognize hyperlinked terms 
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(803) 

V 



(803) 



Vocabulary of 
hyperllnked terms 



"Phenols" 



'Resveratrol" 



"Tannins" 



"Ellagic acid" 



"Hydroxycinnamates" 



"Flavon-3-ois" 



"Free radicals" 



"Anthocyanin" 



"Gallic acid" 




Speaker 
(800) 




workstation 



(802) 



Speech Recognition System 
(801) 




Fig. 8: During the speech, the Speech Recognition System 
recognizes hyperlinked terms pronounced by the speaker 
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Speech Server 




Fig. 9: Hyperlinked terms and associated addresses and univer 
times are recorded on a Speech Hyperlinlc-Time Table on a 
Speech Server 
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(1001) 



Speech Hyperlink-Time Table 



(1002) 



SPEjEfcHJniLE: 
'-mie&HealtFr^ 


SPEAKER: 
Louis Durand 


SPEECHJJYPCnLINK TIMFTABI F, IIRI 


RECOGNITION 

TIME 


HYPERLINKED 
TERM 


HYPERLINK URL 


12/05/2001 14:22:52 


Phenols 


http://\ftAm.stonehillwinery.com/health.htmI 


12/05/2001 14:23:12 


Resveratrol 


http://www.ag.uluc.edu/^ffh/resvera.html 


12/05/2^01 14:23:25 


Tanning 


http://wvvw.mangiarebene,net/wlne/manu^l/health.htrnl 


12/05/2601 14:23:39 


Ellagic s^\d 


http://vmw,hopeforcancerxom/ellagicacid\htm 


12/05/2091 14:23:57 


Hydroxy(3|nnamates 


n up!//www. swets. n 1/ sps/jou rnais/pDoDu 


12/05/20Q1 14:24:11 


Flavon-3-J^ls 


http://w\Aw.nyas.org/scitech/sum/conf_01_(k26.html 


12/05/2001 14:24:35 


Free radicals 


http://vmw. who.sci .eg/publications/emhj/04C^21 .htm 


1 2/05/200l\l 4:24:49 


Anthocyanib 


http://wvvw.orst.edu/dept/lpi/ss01/anthocyanir\^»htrnl 


1 2/05/2001 \l4:25:08 


Gallic acid \ 


http://www.vinsetsante.com/za5e.html \ 
























^ 









(1000) (1003) 



(1004) 



(1005) 



Fig. 10: Example of Speech Hyperlink-Time Table stored on tiie 
Speech Server 
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Fig. 1 1 : During the speech, the auditor selects interesting 
hyperlinked terms by pressing a reserved l<:ey on the auditor device 
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(1202) 



Selections Hyperlink-Time Table 



(1201) 



SPEE&H TITLE: 


SPEAKER: 

Louis Durand < 


SPEECtt-H-YPERttNiesTIMETABfcE4*RL: /^^ 

nttpi//wv\w-air6ciDuy©r.curri/cun^^ i / o-nim/^^ 


/^ELECTION TIME \ 


HYPERUNK 


HYPERLINK URL 


12/05/2001 14:23:14 


\ 




12/05/2001 14:23:42 


) 




12/05/2001 14:24:52 > 


/ 




\ y 



































































(1200) 



Fig. 12: Universal times corresponding to the selection of terms 
by the auditor are stored in a Selections Hyperlink-Time Table 
on the auditor device 
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Speech Server 
(1303) 



Speech 
Hyperlink-Time Table 
(1302) 




Auditor device 
(1300) 



Fig. 13: The auditor updates the Selections Hyperlink-Time 
Table from the Speech Hyperlink-Time Table stored on the 
Speech Server 
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SPEECH TITLE: 

Wine & Health 


SPEAKER: 

Louis Durand 


SPEECH HYPERLINK-TIMETABLE URL- 

http://wvm.directbuyer.com/cx)nference-0173 


SpsECTiOFTniWE^ 


HYPERLINK 


HYPERLINK URL 


'12/05/2001 14:23:14^ 






12/05/2001 14:23:42 


^ 




^12/05/2001 14:24:52.- 


-/ 





SPEECH TITLE: 
Wine & Health 



SPEAKER: 
Louis Durand 



SPEECH HYPERLINK-TIMETABLE URL: 

littp://vyAAW.dlrectbuyer.com/conference"0173.htm/ 



RECOGNITION 
TIME 



HYPERUNKED 
TERM 



HYPERLINK URL 



12/05/2001 14:22:52 



Pfienols 



http:/AAww.stonehilIw!nery.com/health.html 



12/05/2001 14:23:12 



Resveratrol 



http:/yWw.ag.uiuc.edu/'-ffh/resvera.litml 



12/05/2001 14:23:25 



Tannins 



http://\Anww.mangfareb0ne.net/wjne/manual/health.htmi 



12/05/2001 14:23:39 



Ellagic acid 



http-y/vvww.hopeforcancer.com/ellaglcacld.htm 



12/05/2001 14:23:57 



Hydroxycinnamates 



http://www.swets.nl/sps/joumals/pb36000s.html 



12/05/2001 14:24:11 



Flavon-3-ols 



http://www.nyas.org/scitech/sum/conf_01_0426.html 



12/05/2001 14:24:35 



Free radicals 



http://www.who.scLeg/publlcat!ons/emhj/0402/21.htm 



12/05/2001 14:24:49 



Anthocyanin 



http://www.orst.edu/dept/Ipi/ss01/anthocyanln.html 



12/05/2001 14:25:08 



Gallic acid 



http://www.vinsetsante.com/za5e.html 



(1401) 



SPEECH TITLE: 
Wine & Health 


SPEAKER: 
Louis Durand 


SPEECH HYPERLINK-TIMETABLE URL: 

http://Vmw.directbuyer.com/conference-0173.htm/ 


SELECTION TIME 


HYPERLINK 


HYPERLINK URL 


12/05/2001 14:23:14 


Resveratrol 


http:/Awww.ag.u!uc.edu/~ffii/resvera.htmI 


12/05/2001 14:23:42 


Ellagic acid . 


http7/wwWthopeforcancer.com/ellagicacicl.lTtm 


12/05/2001 14:24:52 


Anthocyanin 


tittp://www.orst.edu/dept/lpi/ss01/anthocyanin.html 



(1402) 



Fig. 14: Hyperlinked terms selected by the auditor are identified 
and copied from the Speech Hyperlink-Time Table into the 
Selections Hyperlink-Time Table 
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www.ho peforcancer.com 




Speech Server 
(1507) 



"Ellagic acid" 




hyperlink at: 


(1504) 

/ 





SPEECH TTTLE: 
Wine & Health 


SPEAKER: 
Louis Durand 


SPEECH HYPERLINK-TIMETABLE URU ^ 
tffipyAvvvw.4lInxU>uyerxom/bDnrereiw^ 




SELECTION TIME 


HYPERLINK 


' HYPERLINK URL 




12OS/2001 14:23:14 


Resvamtrol 


httpy/www eg lAic edtV-f (Mosvera Jttml 


'■■mm 


ISADSffiOOl 14^^ 
1&/0Sf2001 141^8 


EOafiie oad 
Anlhopyoflbi 


htlp:flWwwJiopetoroanQerxoffi/o88flieacKJJtttw 
Mlp:AVvMWjonLsdiv«tnMpitesOVAfilliocy9u^ 










•;: U 


(1501) 


(1503) 




Seiections 
IHyperlink-Time^ 
Table 
(1502) 

Auditor device 
(1500) 




Fig. 15: Hyperlinks are triggered using the Selections 
i-lyperlink-Time Table 
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WWW. 



iopeforca ncer.com 

Web Servers 



Speech Server 




Retrieved Web 



P^9®= e[lagicacid.htm^^f^ 



(1601) 




ELLAGICACID 

Ellagfc acid Is a phenolic compound found in plants in the form of 
hydroiyzable tannins called ellagitannins. Blagftannlns are esters 
of glucose with hexahydroxydlphenfc add; when hydrolyzed, they 
yield Ellagic acid, the dilactone of hexahydroxydiphenio add. 
Ellagic acid Is a very stable compound and is readily absorbed 
through the gastrohtestlnal system In mammals, including humans] 

Berries are the most common food sources of Ellaglc add. The 

relative amount of Ellaglc acid In average number of micrograms 

per gram of dry weight fruit extract is highest in red raspberries at 




Received 
Web page 
(1604) 



Auditor device 
(1600) 



Rg. 16: Ttie Web page related to the selected term is received 
iTUTTTthe Web server and displayed on the auditor device 
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Editing the text or a draft text of the speech T 

* 



1701 



Selecting on the speech text a plurality of wordsY^ 
or terms for which hyperlinks must be created 

I 



1702 



Creating a Speech Hyperlink Table for the speech^ ^ 
on the speaker workstation 

' i 



1703 



r 



Assigning a name or a description (preferably 
a brief desc ription) to hyperlinked terms 

* 

Assigning a destination address (URL) 
to access multimedia information 




1704 



1705 



Storing in the Speech Hyperlink Table the assigned: 
- hyperlink name (or description), and 
- the destination address 



i 



1706 



Training with the hyperlinked terms 
a Speech Recognition System 
running on the speaker workstation 



1707 



Fig. 17: Method for creating a Speech Hyperlink Table for a 
speech and for training a Speech Recognition System with 
hyperlinked terms 
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Creating a Speech Hyperlink-Time Table 
for the speech on a Speech Server 

i '- 

Recognizing the pronunciation of hyperlinked 
terms by means of a 
Speech Recognition System 



1801 



1802 



i 



Determining the universal-time 
corresponding to the recognition of the term 



1803 



i 



Creating a record Including 
- the universal time; 
- the name of hyperlinked term; 
the address associated with hyperlinked term 

i 



1804 



Storing the record on a Speech Hyperlink-Time 
Table on the Speech Server 



1805 



Fig. 18: Method for creating a Speech Hyperlink-Time Table 
on a Speech Server and for recognizing the pronunciation 
of hyperlinked terms during a speech 
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Creating a Selections Hyperlink-Time Tabl^ 
for the speech on the auditor device 



1 



r 



1901 



/» ■ >>^1902 

Recording on the Selections Hyperlink-Time ^ 

Table the address of the Speech Hyperlink-Time 

Table on a Speech Server 



Listening to the speech given by the 
speaker 



TV' 



1903 



I 



r ' >^ 

Perceiving a topic of Interest for which 
additional Information or service Is desired 



1904 



I 



Selecting a topic by entering a selection 
command on the auditor device 



1905 



I 



Determining the current universal time by means of 
an universal time device (for Instance a GPS receiver) 
integrated or connected to the auditor device 



:5^1906 



I 



Recording the universal time in the Selections 
Hyperlink-Time Table 



1907 



} 



Fig. 19: l\/Iethod for creating a Selections Hyperlink-Time 
Table and for selecting topics of interest in a speech 
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(Accessing the Speech Server from the auditor devicej ^ 

* 



2001 



C 



Sending to the Speech Server 
the recorded universal time 



2002 



/^Searching In the Speech Hyperlink-Time Tabl^ 

on the Speech Server for the hyperiinlced 
^term active at said recorded universal time 

" ♦ ^2004 



Retrieving from the Speech Hyperllnlc-Time 
Table in the Speech Server the name and the 
address of the selected hyperlinked term 



T 



^ Storing the retrieved hyperlink name and 
destination address (URL) into the corresponding 
entry of the Selections Hyperlink-Time Table 

I 



2005 



Selecting a hyperlink from the Selections 
Hyperlink-Time Table 



I 



^ Activating the hyperlink using a browser 
program In the auditor device 

* 



2006 



2007 



Accessing the information and/or service 

• « ~ 



2008 



Retrieving and displaying the information 
and/or service using the browser program 
on the auditor device 



:>^2009 



Fig. 20 : Method for accessing information related 
to selected terms 
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